Parallel Program Performance Metrics: A Comparison and Validation

There are many metrics designed to assist in the performance debugging of large-scale parallel applications. We describe a new technique, called True Zeroing, that permits direct quantitative comparison of the guidance supplied by these metrics on real applications. We apply this technique to three programs that include both numeric and symbolic applications. We compare three existing metrics: Gprof, Critical Path, and Quartz/NPT, and several new variations. Critical Path provided the best overall guidance, but it was not infallible. We also include a set of recommendations to tool builders based on the experience gained during our case study.